> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.
# CLIP
> Core CLIP model implementation for contrastive learning between images and text
## Overview
The `CLIP` class implements the core CLIP (Contrastive Language-Image Pre-Training) model architecture. It combines a vision encoder and text encoder to learn aligned multimodal representations through contrastive learning.
The model outputs normalized image and text features that can be compared in a shared embedding space using cosine similarity.
## Class Definition
```python theme={null}
from open_clip import CLIP
```
## Initialization Parameters
Dimensionality of the joint embedding space for image and text features.
Configuration object for the vision encoder. Controls architecture (ViT, ResNet, or timm model), layer depth, width, and attention settings.
Configuration object for the text encoder. Specifies transformer architecture, vocabulary size, context length, and pooling strategy.
Use QuickGELU activation (as in original OpenAI models) instead of standard GELU. QuickGELU is less memory efficient but maintains compatibility with OpenAI checkpoints.
Initial value for the learned temperature parameter (logit scale) that controls the sharpness of the similarity distribution.
Optional learnable bias term added to logits. When None, no bias is used.
If True, logit\_scale has shape \[1] instead of \[]. Some training frameworks require explicit dimensions.
Precision for model computations (e.g., torch.float16, torch.bfloat16). Used for mixed precision training.
If True, forward() returns a dictionary with named outputs. If False, returns a tuple.
## Attributes
* **visual**: Vision encoder module (VisionTransformer, ModifiedResNet, or TimmModel)
* **transformer**: Text transformer encoder
* **token\_embedding**: Text token embedding layer
* **positional\_embedding**: Learned positional embeddings for text
* **ln\_final**: Final layer normalization for text features
* **text\_projection**: Projection matrix from text features to joint embedding space
* **logit\_scale**: Learned temperature parameter (exponential of stored value)
* **logit\_bias**: Optional learned bias (if init\_logit\_bias is not None)
* **context\_length**: Maximum text sequence length
* **vocab\_size**: Size of text vocabulary
## Key Methods
### encode\_image
```python theme={null}
def encode_image(self, image: torch.Tensor, normalize: bool = False) -> torch.Tensor:
```
Encodes images into the joint embedding space.
**Parameters:**
* `image`: Image tensor of shape `(batch_size, channels, height, width)`
* `normalize`: If True, L2-normalizes the output features
**Returns:** Image features of shape `(batch_size, embed_dim)`
### encode\_text
```python theme={null}
def encode_text(self, text: torch.Tensor, normalize: bool = False) -> torch.Tensor:
```
Encodes tokenized text into the joint embedding space.
**Parameters:**
* `text`: Tokenized text tensor of shape `(batch_size, context_length)`
* `normalize`: If True, L2-normalizes the output features
**Returns:** Text features of shape `(batch_size, embed_dim)`
### get\_logits
```python theme={null}
def get_logits(self, image: torch.Tensor, text: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
```
Computes similarity logits between image and text features.
**Parameters:**
* `image`: Image tensor
* `text`: Tokenized text tensor
**Returns:** Tuple of (image\_logits, text\_logits) representing similarity scores scaled by temperature
### forward
```python theme={null}
def forward(
self,
image: Optional[torch.Tensor] = None,
text: Optional[torch.Tensor] = None
) -> Union[Dict, Tuple]:
```
Forward pass through the model.
**Parameters:**
* `image`: Optional image tensor
* `text`: Optional tokenized text tensor
**Returns:**
* If `output_dict=True`: Dictionary with keys `image_features`, `text_features`, `logit_scale`, and optionally `logit_bias`
* If `output_dict=False`: Tuple of (image\_features, text\_features, logit\_scale) or (image\_features, text\_features, logit\_scale, logit\_bias)
### forward\_intermediates
```python theme={null}
def forward_intermediates(
self,
image: Optional[torch.Tensor] = None,
text: Optional[torch.Tensor] = None,
image_indices: Optional[Union[int, List[int]]] = None,
text_indices: Optional[Union[int, List[int]]] = None,
stop_early: bool = False,
normalize: bool = True,
normalize_intermediates: bool = False,
intermediates_only: bool = False,
image_output_fmt: str = 'NCHW',
image_output_extra_tokens: bool = False,
text_output_fmt: str = 'NLC',
text_output_extra_tokens: bool = False,
output_logits: bool = False,
output_logit_scale_bias: bool = False,
) -> Dict[str, Union[torch.Tensor, List[torch.Tensor]]]:
```
Forward pass that returns intermediate layer features. Useful for feature extraction, analysis, and distillation.
**Returns:** Dictionary with intermediate features, final features, and optionally logits
### lock\_image\_tower
```python theme={null}
def lock_image_tower(self, unlocked_groups: int = 0, freeze_bn_stats: bool = False):
```
Freezes the image encoder for efficient fine-tuning (LiT-style training).
**Parameters:**
* `unlocked_groups`: Number of layer groups to keep trainable (from the end)
* `freeze_bn_stats`: If True, freezes batch normalization statistics
### lock\_text\_tower
```python theme={null}
def lock_text_tower(self, unlocked_layers: int = 0, freeze_layer_norm: bool = True):
```
Freezes the text encoder for efficient fine-tuning.
**Parameters:**
* `unlocked_layers`: Number of transformer layers to keep trainable (from the end)
* `freeze_layer_norm`: If True, freezes layer normalization parameters
### set\_grad\_checkpointing
```python theme={null}
def set_grad_checkpointing(self, enable: bool = True):
```
Enables or disables gradient checkpointing to reduce memory usage during training at the cost of computation.
## Usage Example
```python theme={null}
import torch
from open_clip import CLIP, CLIPVisionCfg, CLIPTextCfg
# Define model configuration
vision_cfg = CLIPVisionCfg(
layers=12,
width=768,
patch_size=16,
image_size=224
)
text_cfg = CLIPTextCfg(
context_length=77,
vocab_size=49408,
width=512,
heads=8,
layers=12
)
# Initialize model
model = CLIP(
embed_dim=512,
vision_cfg=vision_cfg,
text_cfg=text_cfg,
output_dict=True
)
# Forward pass
images = torch.randn(4, 3, 224, 224)
texts = torch.randint(0, 49408, (4, 77))
output = model(images, texts)
image_features = output['image_features'] # (4, 512)
text_features = output['text_features'] # (4, 512)
logit_scale = output['logit_scale'] # scalar
# Compute similarity
similarity = (image_features @ text_features.T) * logit_scale
# Encode separately
image_emb = model.encode_image(images, normalize=True)
text_emb = model.encode_text(texts, normalize=True)
```
## Fine-tuning Example
```python theme={null}
# Lock image tower for text-only fine-tuning (LiT)
model.lock_image_tower(unlocked_groups=0, freeze_bn_stats=True)
# Enable gradient checkpointing for memory efficiency
model.set_grad_checkpointing(enable=True)
# Only text parameters will be updated during training
for name, param in model.named_parameters():
print(f"{name}: requires_grad={param.requires_grad}")
```
## Related
* [CustomTextCLIP](/api/custom-text-clip) - Variant with separately built text tower
* [CoCa](/api/coca-model) - Contrastive Captioner model
* [ClipLoss](/api/clip-loss) - Contrastive loss function for CLIP